Papers with memory overhead
Punctuation-Steered Representation Fine-Tuning (2026.acl-short)
Copied to clipboard
| Challenge: | Existing methods for parameter-efficient fine-tuning (PeFT) are limited due to their prohibitive size and computational demands. |
| Approach: | They propose a method that fine-tunes punctuation representations to achieve performance improvements. |
| Outcome: | The proposed method improves performance by altering the representation space alone . but it results in suboptimal performance due to the effects of the method on the output . |
LoRAPrune: Structured Pruning Meets Low-Rank Parameter-Efficient Fine-Tuning (2024.findings-acl)
Copied to clipboard
| Challenge: | Low-rank adaption (LoRA) is a low-level pruning method that can be expensive and slow to deploy. |
| Approach: | They propose a low-rank adaption pruning framework that provides an accurate structured pruned model in a memory-efficient manner. |
| Outcome: | The proposed pruning framework reduces perplexity and memory usage by 52.6% on LLaMA and T5 models while reducing memory usage. |
HqeKV: Towards Hybrid Quantization and Eviction for KV Cache in Long-Context LLM Inference (2026.findings-acl)
Copied to clipboard
| Challenge: | autoregressive inference requires repeated computation across transformer layers. |
| Approach: | They propose a hybrid compression framework built on both quantization and eviction . they propose varying importance metric and flexible conversion policies to reduce memory overhead . |
| Outcome: | The proposed framework outperforms state-of-the-art methods under memory constraints. |
FlexQuant: A Flexible and Efficient Dynamic Precision Switching Framework for LLM Quantization (2025.findings-emnlp)
Copied to clipboard
Fangxin Liu, Zongwu Wang, Jinhong Xia, Junping Zhao, Shouren Zhao, Jinjin Li, Jian Liu, Li Jiang, Haibing Guan
| Challenge: | Existing methods for quantization of large language models struggle to adapt to dynamic workloads. |
| Approach: | a new framework optimizes the trade-off between inference speed and accuracy . FlexQuant enables fine-grained, layer-wise mixed-precision quantization . |
| Outcome: | a new framework optimizes the trade-off between inference speed and accuracy . it achieves a 1.3 speedup across diverse language tasks with negligible accuracy loss . |
Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Models (2022.findings-acl)
Copied to clipboard
| Challenge: | Prompting language models (LMs) with training examples and task descriptions has been seen as critical to recent successes in few-shot learning. |
| Approach: | They propose to fine tune masked language models with training examples and task descriptions to reduce prompt engineering by using null prompts. |
| Outcome: | The proposed prompts can be used to improve few-shot learning by finetuning only the bias terms while updating only 0.1% of the parameters. |
Representation Interventions Enable Lifelong Knowledge Memory Control in LLMs (2026.acl-long)
Copied to clipboard
Xuyuan Liu, Shengyu Chen, Xinshuai Dong, Yanchi Liu, Xujiang Zhao, Haoyu Wang, Yujun Yan, Haifeng Chen, Zhengzhang Chen
| Challenge: | Large language models (LLMs) produce outdated or inaccurate content. Updating their knowledge efficiently and accurately without costly retraining is a major challenge. |
| Approach: | They propose a robust and scalable method that treats knowledge control as interventions within the model’s representation space. |
| Outcome: | The proposed method achieves fine-grained control over complex, unstructured knowledge while maintaining general utility with frozen base weights. |
CONGRAD: Conflicting Gradient Filtering for Multilingual Preference Alignment (2026.eacl-long)
Copied to clipboard
Jiangnan Li, Thuy-Trang Vu, Christian Herold, Amirhossein Tebbifakhr, Shahram Khadivi, Gholamreza Haffari
| Challenge: | Naive joint training of large language models can suffer from negative interference. |
| Approach: | They propose a filtering method that aggregates cross-lingually beneficial gradients and filters for those with high cross-linguistic affinity. |
| Outcome: | The proposed method outperforms baselines in both seen and unseen languages with minimal alignment tax. |
Fast Randomized Low-Rank Adaptation of Pre-trained Language Models with PAC Regularization (2024.findings-acl)
Copied to clipboard
| Challenge: | Low-rank adaptation (LoRA) is an efficient way to fine-tune large language models (LLMs) but its memory overhead restricts it to scale up when the model size increases. |
| Approach: | They propose a low-rank adaptation method which decomposes model weight updates into a pair of low-ranked projection matrices and incorporates a regularizer to improve generalization. |
| Outcome: | The proposed method achieves better performance under few-shot settings and similar performance to the SOTA low-rank adaptation methods. |
Boundary-Guided Policy Optimization for Memory-efficient RL of Diffusion Large Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for reinforcement learning (RL) require a large sample size to be implemented. |
| Approach: | They propose a memory-efficient RL algorithm that maximizes a lower bound of the ELBO-based objective. |
| Outcome: | Experiments show that BGPO outperforms previous RL algorithms for diffusion large language models in math problem solving, code generation, and planning tasks. |
Extremely Low Bit Transformer Quantization for On-Device Neural Machine Translation (2020.findings-emnlp)
Copied to clipboard
Insoo Chung, Byeongwook Kim, Yoonjung Choi, Se Jung Kwon, Yongkweon Jeon, Baeseong Park, Sangha Kim, Dongsoo Lee
| Challenge: | Quantization is an effective technique to address heavy computation load and memory overhead during inference. |
| Approach: | They propose a low-bit quantization strategy to represent Transformer weights by an extremely low number of bits. |
| Outcome: | The proposed model achieves 11.8 smaller model size than baseline model, with less than -0.5 BLEU. |
Fixed Input Parameterization for Efficient Prompting (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent studies have shown that attaching prompts to the input is effective at conditioning Language Models (LMs) however, prompts are always included in the input text during inference, thus incurring substantial computational and memory overhead. |
| Approach: | They propose to inject a fixed prompt into the parameters of an LM to be an efficient alternative to attaching fixed prompts to the input. |
| Outcome: | The proposed method can be up to 280 times more efficient in terms of FLOPs than previous approaches. |
Accurate KV Cache Quantization with Outlier Tokens Tracing (2025.acl-long)
Copied to clipboard
Yi Su, Yuechi Zhou, Quantong Qiu, Juntao Li, Qingrong Xia, Ping Li, Xinyu Duan, Zhefeng Wang, Min Zhang
| Challenge: | Large Language Models (LLMs) require substantial computational resources during deployment. |
| Approach: | They propose a method to identify outlier tokens and exclude them from quantization . they find that the method can deliver a 6.4 times reduction in memory usage and a 2.5 times increase in throughput . |
| Outcome: | The proposed method delivers a 6.4 times reduction in memory usage and a 2.5 times increase in throughput under 2-bit quantization. |
Variator: Accelerating Pre-trained Models with Plug-and-Play Compression Modules (2023.findings-emnlp)
Copied to clipboard
Chaojun Xiao, Yuqi Luo, Wenbin Zhang, Pengle Zhang, Xu Han, Yankai Lin, Zhengyan Zhang, Ruobing Xie, Zhiyuan Liu, Maosong Sun, Jie Zhou
| Challenge: | Large language models (LLMs) have been successful on NLP tasks but require huge parameter sizes and computational resources. |
| Approach: | They propose a parameter-efficient acceleration method that enhances computational efficiency through plug-and-play compression plugins. |
| Outcome: | The proposed method saves 53% computational costs using only 0.9% additional parameters with a performance drop of less than 2%. |
LLoCO: Learning Long Contexts Offline (2024.emnlp-main)
Copied to clipboard
Sijun Tan, Xiuyu Li, Shishir G Patil, Ziyang Wu, Tianjun Zhang, Kurt Keutzer, Joseph Gonzalez, Raluca Popa
| Challenge: | Large language models are still unable to handle long contexts due to the quadratic computational and memory overhead of the self-attention mechanism and the substantial KV cache sizes during generation. |
| Approach: | They propose a method to learn contexts offline through context compression and in-domain parameter-efficient finetuning with LoRA. |
| Outcome: | The proposed model outperforms in-context learning while using 30 fewer tokens during inference and significantly reduces the cost of long document question answering. |
Deputy: Accelerating Large Language Model Inference with Dynamic Low-Rank Substitution (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing dynamic schemes such as early-exit and layer-drop reduce FLOPs but break batch processing or introduce KV-cache inconsistency. |
| Approach: | They propose a dynamic low-rank substitution framework that employs a lightweight decision module at each layer to dynamically determine the execution branch for different tokens. |
| Outcome: | The proposed model reduces computation by approximately 40% compared to the original dense model while outperforming existing baseline methods. |
CoViPAL: Layer-wise Contextualized Visual Token Pruning for Large Vision-Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to prune redundant vision tokens struggle in shallow layers due to the lack of contextual information. |
| Approach: | They propose a layer-wise contextualized visual token pruning method that uses a plug-and-play Pruning Module to prune redundant vision tokens. |
| Outcome: | The proposed method outperforms training-free pruning methods under equal token budgets and surpasses training based methods with comparable supervision. |
Memory-Efficient Fine-Tuning of Transformers via Token Selection (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning require caching of intermediate activations to update weights during the backward pass. |
| Approach: | They develop a method to reduce memory usage in fine-tuning of transformers by backpropagating through just a subset of input tokens. |
| Outcome: | The proposed method reduces memory usage and memory footprint on large transformer models . it can be easily combined with existing methods like LoRA, reducing memory cost . |
Unifying Uniform and Binary-coding Quantization for Accurate Compression of Large Language Models (2025.acl-long)
Copied to clipboard
Seungcheol Park, Jeongin Bae, Beomseok Kwon, Minjun Kim, Byeongwook Kim, Se Jung Kwon, U Kang, Dongsoo Lee
| Challenge: | Quantization is essential for deploying large language models (LLMs) efficiently since they require expensive computational and memory costs. |
| Approach: | They propose a quantization method that unifies flexible mapping techniques to optimize parameters precisely. |
| Outcome: | The proposed method outperforms existing methods and achieves higher accuracy on GSM8K benchmark. |
HyperAdaLoRA: Accelerating LoRA Rank Allocation During Training via Hypernetworks without Sacrificing Performance (2026.findings-acl)
Copied to clipboard
Hao Zhang, Zhenjia Li, Yifan Gao, Xi Xiao, Heng Zhang, Shuyang Zhang, null Xiaoxincc, Bo Huang, Yuhang Wu, Tianyang Wang, Hao Xu
| Challenge: | Low-Rank Adaptation (LoRA) assumes a uniform rank r for each incremental matrix, not accounting for the varying significance of weight matrices across modules and layers. |
| Approach: | They propose a framework that allows for faster convergence of low-rank adaptive models . they use a hypernetwork to prune the outputs of the hypernetworks to generate parameters . |
| Outcome: | The proposed framework accelerates convergence of AdaLoRA by leveraging a hypernetwork. |
Universally Empowering Zeroth-Order Optimization via Adaptive Layer-wise Sampling (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning Large Language Models are slow and lack of performance. |
| Approach: | They propose a Zeroth-Order optimization framework that uses forward passes to fine-tune Large Language Models. |
| Outcome: | The proposed framework achieves 1.7 to 3.0 wall-clock acceleration on LLaMA and OPT models. |
Unlocking the Edge deployment and ondevice acceleration of multi-LoRA enabled one-for-all foundational LLM (2026.findings-acl)
Copied to clipboard
Sravanth Kodavanti, Sowmya Vajrala, Srinivas Soumitri Miriyala, Utsav Tiwari, Uttam Kumar, Utkarsh Kumar Mahawar, Achal Pratap Singh, Arya D, Narendra Mutyala, Vikram Nelvoy Rajendiran, Sharan Kumar Allur, Euntaik Lee, Dohyoung Kim, HyeonSu Lee, Gyusung Cho, JungBae Kim
| Challenge: | a framework for efficient on-device inference of large language models is needed for smartphones . memory, latency, and runtime flexibility are constraints for large language model deployments. |
| Approach: | They propose a hardware-aware framework for efficient on-device inference of a LLaMA-based multilingual foundation model for Samsung Galaxy S24 and S25 devices with SM8650 and SM8750 chipsets respectively. |
| Outcome: | The proposed framework improves memory, latency and performance across 9 languages and 8 tasks. |
LR-DWM: Efficient Watermarking for Diffusion Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Current methods for large language models rely on tokens being generated sequentially . left-right Diffusion watermarking uses a fixed, deterministic left-to-right order . |
| Approach: | They propose a scheme that biases tokens based on both left and right neighbors . left-Right Diffusion Watermarking is a low-latency alternative to autoregressive models . |
| Outcome: | The proposed method can be watermarked efficiently with minimal runtime and memory overhead. |